Image processing apparatus, image capture apparatus, and control method for adding an effect of a virtual light source to a subject

ABSTRACT

With respect to a subject included in an image, the illuminating condition by an ambient light source in an environment where the image was captured is estimated, and based on the estimation result, the effect of a virtual light source that was non-existent at the time of image capture is computed. More specifically, the effect of the virtual light source is computed using an illumination direction of the virtual light source and the reflective characteristics of the subject illuminated by the virtual light source, which have been determined based on the estimation result, and an image derived from addition of the effect of the virtual light source is output.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/203,914, filed on Nov. 29, 2018, which is a continuation of U.S.patent application Ser. No. 15/432,412, filed on Feb. 14, 2017, now U.S.Pat. No. 10,171,744, which claims the benefit of and priority toJapanese Patent Application No. 2016-029134, filed on Feb. 18, 2016,each of which is hereby incorporated by reference herein in theirentirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing apparatus, an imagecapture apparatus, and a control method, and particularly to an imageprocessing technique for adding, to an image, an illumination effect ofa virtual light source that is supplementarily defined after shooting.

Description of the Related Art

There is a technique for reducing shadows cast on a subject by adding,to an image obtained after shooting, an illumination effect of a virtuallight source that was non-existent in a shooting environment. JapanesePatent Laid-Open No. 2010-135996 describes a technique for detecting aface region in an image, and adjust the brightness using a definedvirtual light source with respect to the face region and a shadow regionwith luminance lower than the average luminance of the face region.

Although the technique described in Japanese Patent Laid-Open No.2010-135996 defines the tints, intensity distribution, and direction ofthe virtual light source to adjust the brightness with respect to atarget subject, this technique has the possibility that a favorableillumination effect may not be added in consideration of the environmentin which the image was shot and the subject. For example, this techniquehas the possibility that gloss and luster, as well as a stereoscopiceffect with an enhanced contrast, may not be favorably represented.

SUMMARY OF THE INVENTION

The present invention was made in view of such problems in theconventional technique. The present invention provides an imageprocessing apparatus, an image capture apparatus, and a control methodthat generate an image to which a favorable illumination effect has beenadded in accordance with a scene and a subject after image capture.

The present invention in its first aspect provides an n image processingapparatus, comprising: an obtainment unit configured to obtain an imagederived from image capture; a computation unit configured to compute aneffect of a virtual light source on a subject included in the imageobtained by the obtainment unit, the virtual light source beingnon-existent at the time of the image capture; and an output unitconfigured to output an image derived from addition of the effect of thevirtual light source to the subject based on a result of the computationby the computation unit, wherein the computation unit includes: anestimation unit configured to, based on the obtained image, estimate anilluminating condition by an ambient light source in an environmentwhere the image was captured, a determination unit configured to, basedon a result of the estimation by the estimation unit, determine anillumination direction of the virtual light source and reflectivecharacteristics of the subject illuminated by the virtual light source,and a processing unit configured to compute the effect of the virtuallight source based on the illumination direction of the virtual lightsource and the reflective characteristics of the subject determined bythe determination unit.

The present invention in its second aspect provides an image processingapparatus, comprising: an obtainment unit configured to obtain an imagederived from image capture; a computation unit configured to compute aneffect of a virtual light source on a subject included in the imageobtained by the obtainment unit, the virtual light source beingnon-existent at the time of the image capture; and an output unitconfigured to output an image derived from addition of the effect of thevirtual light source to the subject based on a result of the computationby the computation unit, wherein the computation unit includes: adetermination unit configured to, based on a shooting scene of theobtained image or a type of the subject, determine an illuminationdirection of the virtual light source and reflective characteristics ofthe subject illuminated by the virtual light source, and a processingunit configured to compute the effect of the virtual light source basedon the illumination direction of the virtual light source and thereflective characteristics of the subject determined by thedetermination unit.

The present invention in its third aspect provides an image captureapparatus, comprising: an image capture unit configured to generate animage through image capture; and an image processing apparatus thatobtains the image generated by the image capture unit and outputs animage derived from addition of an effect of a virtual light source,wherein the image processing apparatus comprising: an obtainment unitconfigured to obtain an image derived from image capture; a computationunit configured to compute the effect of the virtual light source on asubject included in the image obtained by the obtainment unit, thevirtual light source being non-existent at the time of the imagecapture; and an output unit configured to output an image derived fromaddition of the effect of the virtual light source to the subject basedon a result of the computation by the computation unit, wherein thecomputation unit includes an estimation unit configured to, based on theobtained image, estimate an illuminating condition by an ambient lightsource in an environment where the image was captured, a determinationunit configured to, based on a result of the estimation by theestimation unit, determine an illumination direction of the virtuallight source and reflective characteristics of the subject illuminatedby the virtual light source, and a processing unit configured to computethe effect of the virtual light source based on the illuminationdirection of the virtual light source and the reflective characteristicsof the subject determined by the determination unit.

The present invention in its fourth aspect provides an image processingapparatus, comprising: an evaluation unit configured to evaluate a stateof light illumination of a subject included in a shot image, the lightillumination being performed by an ambient light source at the time ofshooting; and a generation unit configured to generate a corrected imageby applying, to the shot image, image processing for adding an effect oflight illumination performed by a virtual light source based on theevaluation by the evaluation unit, the virtual light source beingnon-existent at the time of the shooting, wherein in applying the imageprocessing, the generation unit controls an arrangement of the virtuallight source and reflective characteristics of the subject under thelight illumination performed by the virtual light source based on theevaluation by the evaluation unit.

The present invention in its fifth aspect provides a control method foran image processing apparatus, the control method comprising: obtainingan image derived from image capture; computing an effect of a virtuallight source on a subject included in the obtained image, the virtuallight source being non-existent at the time of the image capture; andoutputting an image derived from addition of the effect of the virtuallight source to the subject based on a result of the computing, whereinthe computing includes: based on the obtained image, estimating anilluminating condition by an ambient light source in an environmentwhere the image was captured, based on a result of the estimation in theestimating, determining an illumination direction of the virtual lightsource and reflective characteristics of the subject illuminated by thevirtual light source, and executing processing for computing the effectof the virtual light source based on the determined illuminationdirection of the virtual light source and the determined reflectivecharacteristics of the subject.

The present invention in its sixth aspect provides a control method foran image processing apparatus, the control method comprising: evaluatinga state of light illumination of a subject included in a shot image, thelight illumination being performed by an ambient light source at thetime of shooting; and generating a corrected image by applying, to theshot image, image processing for adding an effect of light illuminationperformed by a virtual light source based on the evaluation in theevaluating, the virtual light source being non-existent at the time ofthe shooting, wherein in applying the image processing in thegenerating, an arrangement of the virtual light source and reflectivecharacteristics of the subject under the light illumination performed bythe virtual light source are controlled based on the evaluation in theevaluating.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a functional configuration of adigital camera 100 according to one or more embodiments of the presentinvention.

FIG. 2 is a block diagram showing a detailed configuration of an imageprocessing unit 105 according to one or more embodiments of the presentinvention.

FIG. 3 is a block diagram showing a detailed configuration of are-lighting processing unit 114 according to one or more embodiments ofthe present invention.

FIG. 4 is a diagram for describing the operations of a normalcalculation unit 310 according to one of more embodiments of the presentinvention.

FIG. 5 is a flowchart exemplarily showing parameter determinationprocessing executed by the digital camera 100 according to a firstembodiment of the present invention.

FIGS. 6A and 6B are diagrams for describing the estimation of thecharacteristics of an ambient light source according to one or moreembodiments of the present invention.

FIG. 7 shows examples of determination criteria for various virtuallight source parameters according to the first embodiment of the presentinvention.

FIGS. 8A, 8B, 8C, and 8D are diagrams for describing various virtuallight source parameters according to the first embodiment of the presentinvention.

FIGS. 9A, 9B, 9C, 9D, 9E, 9F, 9G, and 9H show examples of images beforeand after re-lighting processing according to the first embodiment ofthe present invention.

FIG. 10 is a flowchart exemplarily showing parameter determinationprocessing executed by the digital camera 100 according to a secondembodiment of the present invention.

FIG. 11 shows examples of determination criteria for various virtuallight source parameters according to the second embodiment of thepresent invention.

FIGS. 12A, 12B, and 12C are diagrams for describing various virtuallight source parameters according to the second embodiment of thepresent invention.

FIG. 13 is a flowchart exemplarily showing parameter determinationprocessing executed by the digital camera 100 according to a thirdembodiment of the present invention.

FIGS. 14A and 14B are diagrams for describing various virtual lightsource parameters according to the third embodiment of the presentinvention.

DESCRIPTION OF THE EMBODIMENTS First Embodiment

The following describes an exemplary embodiment of the present inventionin detail with reference to the drawings. Note that the embodimentdescribed below presents an example in which the present invention isapplied to a digital camera—one example of an image processingapparatus—that can execute re-lighting processing for adding anillumination effect of a virtual light source. However, an image capturefunction is not an essential element in embodying the present invention,and the present invention is applicable to any apparatus that canexecute various types of processing for adding an illumination effect ofa supplementary virtual light source to image signals. In the presentspecification, “re-lighting processing” denotes processing forsupplementarily defining a virtual light source that was non-existent inthe actual shooting environment and adding an illumination effect of thevirtual light source with respect to image signals obtained throughimage capture.

<<Configuration of Digital Camera 100>>

FIG. 1 is a block diagram showing a functional configuration of adigital camera 100 according to one or more embodiments of the presentinvention.

A lens group 101 is an image capture optical system including a zoomlens and a focusing lens, and directs light beams reflected by a subjectin a shooting environment and light beams from light sources to an imagecapture unit 103. A shutter 102 having diaphragm functions controls thelight amounts of directed light beams, and the image capture unit 103converts the resultant light beams into analog image signals. The imagecapture unit 103 is, for example, a CCD, a CMOS sensor, or another imagecapture apparatus; the image capture unit 103 photoelectrically convertsan optical image formed on an imaging plane, and outputs the resultantanalog image signals. An A/D converter 104 converts the analog imagesignals output from the image capture unit 103 into digital imagesignals (image data), and an image processing unit 105 applies varioustypes of image processing, such as white balance processing, gammacorrection processing, edge enhancement processing and color correctionprocessing, to the digital image signals (image data). A face detectionunit 113 executes face detection processing for detecting a face regionof a human included in input image data, and outputs a detection result.A re-lighting processing unit 114 applies re-lighting processing foradding an illumination effect of a virtual light source to input imagedata. In various types of image-related processing, an image memory 106is usable as a working area and a data storage area, and a memorycontrol unit 107 controls the overall operations related to access(e.g., writing and reading) to the image memory 106.

The details of the image processing unit 105 and re-lighting processingunit 114 will be described later using other drawings. In the followingdescription, each of various types of image processing executed by thedigital camera 100 according to the present embodiment is discretelyconfigured in correspondence with the image processing unit 105, facedetection unit 113, or re-lighting processing unit 114. However, itshould be easily understood that, in one or more embodiments of thepresent invention, these various types of image processing can beconfigured in correspondence with an arbitrary number of processingcircuits, where the arbitrary number is one or more.

Once various types of image processing and overlap processing have beenapplied to image data obtained through image capture, a D/A converter108 converts the image data into analog image signals for display. Adisplay unit 109 (e.g., an LCD) displays the analog image signals fordisplay, thereby providing the functions of an electronic viewfinder toa photographer. In recording (storing) image data obtained through ashooting operation, a codec unit 110 converts the image data into datafor recording in accordance with a preset compression and encodingformat, and the data for recording is recorded to a recording medium112, which is a built-in memory, a memory card or another recordingapparatus, via an I/F 111. In displaying image data recorded in therecording medium 112, the codec unit 110 decodes the image data.

A system control unit 50 is, for example, a CPU or another computationapparatus, and controls the operations of blocks included in the digitalcamera 100. More specifically, the system control unit 50 controls theoperations of the blocks by reading out operation programs of the blocksstored in a nonvolatile memory 121, extracting the operation programs toa system memory 122, and executing the operation programs. Thenonvolatile memory 121 is, for example, an EEPROM or another nonvolatilestorage apparatus. The nonvolatile memory 121 stores, for example,various types of parameters necessary for the operations of the blocks,in addition to the operation programs of the blocks. The system memory122 may be a rewritable volatile memory, and is used not only as an areato which the operation programs of the blocks are extracted, but also asa storage area that temporarily holds, for example, intermediate datathat is output along with the operations of the blocks.

The system control unit 50 also performs exposure control and rangingcontrol (control over the states of the lens group 101, shutter 102, andimage capture unit 103) related to image capture based on predeterminedevaluation values that have been generated by the image processing unit105 using the result of face detection by the face detection unit 113and image data obtained through image capture. As a result,through-the-lens (TTL) autofocus (AF) processing, auto exposure (AE)processing, auto white balance (AWB) processing, and the like arerealized.

The digital camera 100 also includes a flash 123 that functions as areal light source at the time of shooting, a ranging sensor 124 thatmeasures a distance between the digital camera 100 and the subject, anda console unit 120 serving as a user interface that detects varioustypes of operational input. When the console unit 120 detects that anoperation input has been made, it outputs a corresponding control signalto the system control unit 50.

In the following description of the present embodiment, processing isrealized by circuits and processors corresponding to blocks that areincluded in the digital camera 100 as items of hardware. However, thepresent invention is not limited to being embodied in this way, andprocessing of the blocks may be realized by programs that executeprocessing similar to the processing of the blocks.

<Configuration of Image Processing Unit 105>

A configuration of the image processing unit 105 will now be describedin detail using a block diagram of FIG. 2. It will be assumed that,along with image capture by the digital camera 100 according to thepresent embodiment, the image processing unit 105 receives, as input, adigital image signal of a Bayer format that indicates a signal level ofone of R, G, and B components for each pixel.

A development processing unit 200 applies development processing to theinput image signal of the Bayer format to interpolate color componentswithout signal levels for each pixel of the image signal, therebygenerating image signals of R, G, and B components (RGB signals). A WBamplification unit 201 adjusts white balance (WB) by amplifying thesignal level of each color component of the RGB signals based on a WBgain value determined by the system control unit 50. A luminance andcolor signal generation unit 202 generates a luminance signal (Y signal)from the WB-adjusted RGB signals, and outputs the luminance signal (Ysignal) to an edge enhancement processing unit 203. The luminance andcolor signal generation unit 202 also outputs the input RGB signals to acolor conversion processing unit 205.

The edge enhancement processing unit 203 applies edge enhancementprocessing to the input Y signal, and outputs the resultant Y signal toa luminance gamma processing unit 204. The luminance gamma processingunit 204 applies gamma correction processing to the Y signal derivedfrom the edge enhancement processing, and outputs the corrected Y signalto the image memory 106 via the memory control unit 107.

On the other hand, the color conversion processing unit 205 appliespredetermined matrix computation to the input RGB signals to change thesignal levels of the color components to achieve a preset color balance.The color conversion processing unit 205 outputs the changed RGB signalsto a color gamma processing unit 206 and an evaluation value obtainmentunit 208. The color gamma processing unit 206 applies gamma correctionprocessing to the color components of the RGB signals that have beenchanged to achieve the preset color balance, and outputs the resultantRGB signals to a chrominance signal generation unit 207. The chrominancesignal generation unit 207 generates chrominance signals R-Y and B—Ybased on the input RGB signals, and outputs the chrominance signals R-Yand B-Y to the image memory 106 via the memory control unit 107.

It will be assumed that image data for recording is formed as luminanceand chrominance signals, that is, Y, R-Y, and B-Y signals in the digitalcamera 100 according to the present embodiment. Specifically, along witha shooting operation, Y, R-Y, and B-Y signals stored in the image memory106 are compressed and encoded by the codec unit 110 and recorded to therecording medium 112.

The evaluation value obtainment unit 208 obtains and outputs information(evaluation values) for analyzing how the subject is illuminated(illumination state) by a light source that really existed in a shootingenvironment (ambient light source). In later-described re-lightingcontrol processing executed by the digital camera 100 according to thepresent embodiment, the evaluation value obtainment unit 208 obtains thefollowing as the evaluation values: information of an average luminanceof the subject, and information of luminance histograms that are inone-to-one correspondence with horizontal and vertical lines in a faceregion. It is sufficient to store the output information of theevaluation values to, for example, the system memory 122.

<Configuration of Re-Lighting Processing Unit 114>

A configuration of the re-lighting processing unit 114 will now bedescribed in detail using a block diagram of FIG. 3. In the followingdescription, in a preset mode for applying the re-lighting processing,the re-lighting processing unit 114 of the digital camera 100 accordingto the present embodiment applies the processing to luminance andchrominance signals generated by the image processing unit 105.Alternatively, the re-lighting processing unit 114 may apply theprocessing to luminance and chrominance signals that have been read outfrom the recording medium 112 and extracted to the image memory 106.

An RGB signal conversion unit 301 converts input luminance andchrominance signals into RGB signals by applying predetermined colorconversion processing thereto, and outputs the RGB signals to a de-gammaprocessing unit 302. Based on the gamma characteristics of gammacorrection that has been applied to the input RGB signal, the de-gammaprocessing unit 302 executes de-gamma processing for eliminating theeffect of the gamma correction. At the time of image capture, thede-gamma processing unit 302 converts the input RGB signals intopre-gamma correction linear signals using the inverse of the gammacharacteristics of the gamma correction that has been applied by thecolor gamma processing unit 206. After applying the de-gamma processing,the de-gamma processing unit 302 outputs the RGB signals (R_(t), G_(t),B_(t)) to a virtual light source addition processing unit 303 and alater-described virtual light source reflection component calculationunit 311.

The virtual light source addition processing unit 303 applies there-lighting processing for adding the effect of a virtual light sourceto the RGB signals derived from the de-gamma processing executed by thede-gamma processing unit 302. The effect of the virtual light source isadded by adding RGB signals for addition (R_(a), G_(a), B_(a)) outputfrom the virtual light source reflection component calculation unit 311.

The virtual light source reflection component calculation unit 311calculates, for each color component, the effect exerted on the subjectby placing the virtual light source. The virtual light source reflectioncomponent calculation unit 311 according to the present embodimentcalculates the effect exerted when the virtual light source is placed toemit follow light or oblique light. Below, a description is given ofprocessing of the blocks executed in relation to the generation of theRGB signals for addition by the virtual light source reflectioncomponent calculation unit 311 when the virtual light source is placedto emit follow light/oblique light.

A normal calculation unit 310 calculates a normal map for the subjectbased on subject distance information that has been obtained by theranging sensor 124 along with image capture and that indicates adistribution of distances between the digital camera 100 and thesubject. Any known technique may be used to generate the normal mapbased on the subject distance information; the following describes anexemplary method of generating the normal map using FIG. 4.

FIG. 4 shows a positional relationship between the digital camera 100and a subject 401 on a plane formed by an optical axis of the digitalcamera 100 and a crosswise (horizontal) direction of an image sensor atthe time of image capture. Provided that the subject 401 is shaped asshown in this figure, normal information 403 of a region 402 of thesubject is obtained from a horizontal difference ΔH in the region 402 ona captured image and from a distance-wise (depth-wise) difference ΔD inthe region 402 obtained from the subject distance information. Morespecifically, the normal 403 can be defined based on gradientinformation of the region 402 calculated based on ΔH and ΔD. Based onthe subject distance information input in correspondence with thecaptured image, the normal calculation unit 310 forms the normal map bycalculating pieces of normal information in one-to-one correspondencewith pixels of the captured image, and outputs the normal map to thevirtual light source reflection component calculation unit 311.

The virtual light source reflection component calculation unit 311calculates the effect exerted by illuminating the subject with thevirtual light source to be defined based on various virtual light sourceparameters and the following elements related to the subject: a distanceK to the light source, normal information N, a specular reflectiondirection S, and a reflectance k. That is, the virtual light sourcereflection component calculation unit 311 calculates reflectioncomponents associated with the virtual light source that are reflectedwhen the virtual light source illuminates the subject and are incidenton the digital camera 100. Then, the virtual light source reflectioncomponent calculation unit 311 outputs the RGB signals (R_(a), G_(a),B_(a)) derived from computation of the reflection components to thevirtual light source addition processing unit 303.

For example, in order to define a virtual point light source 404 shownin FIG. 4 as the virtual light source, the virtual light sourcereflection component calculation unit 311 calculates reflectioncomponents associated with the virtual light source that are derivedfrom diffuse reflection and specular reflection of light emitted by thepoint light source on a surface of the subject. For simplicity, theexample of FIG. 4 will be described below without taking intoconsideration a height direction (a direction of a normal to a planeformed by the horizontal direction and a direction of the optical axis(distance) in FIG. 4); however, it is understood that three-dimensionalvectors may be taken into consideration in the calculation. For example,a value of a diffuse reflection component that arises on the subject ata horizontal pixel position H1 on the captured image is proportional tothe inner product of a normal N1 at a corresponding position in thenormal map and a direction vector L1 of a light beam reaching thesubject from the virtual light source, and is inversely proportional tothe square of a distance K1 between the virtual light source and thesubject. On the other hand, a value of a specular reflection component,for which contribution to incidence on the digital camera 100 needs tobe taken into consideration, is proportional to the inner product of aspecular reflection direction S1 determined by the normal N1 and thedirection vector L1 and a direction V1 from the subject toward thedigital camera 100. In view of the above, the reflection components(R_(a), G_(a), B_(a)) that arise on the subject in association with thedefined virtual light source can be expressed as follows using the RGBsignals (R_(t), G_(t), B_(t)) derived from the de-gamma processing:

$R_{a} = {\sum\limits_{Lights}{\left\{ {{\alpha \times \left\{ {k_{d} \times \frac{\left( {{- L} \cdot N} \right)}{K^{2}}} \right)} + {k_{s} \times \left( {S \cdot V} \right)^{\beta}}} \right\} \times R_{w} \times R_{t}}}$$G_{a} = {\sum\limits_{Lights}{\left\{ {{\alpha \times \left\{ {k_{d} \times \frac{\left( {{- L} \cdot N} \right)}{K^{2}}} \right)} + {k_{s} \times \left( {S \cdot V} \right)^{\beta}}} \right\} \times 1 \times G_{t}}}$$B_{a} = {\sum\limits_{Lights}{\left\{ {{\alpha \times \left\{ {k_{d} \times \frac{\left( {{- L} \cdot N} \right)}{K^{2}}} \right)} + {k_{s} \times \left( {S \cdot V} \right)^{\beta}}} \right\} \times B_{w} \times B_{t}}}$

Where, virtual light source parameters is represented by the intensity αand control values for the colors (tints) of the light source (red colorcomponent R_(w) and blue color component B_(w)). Furthermore, k_(d) andk_(s) respectively represent the diffuse reflectance and specularreflectance of the subject, L represents a direction vector toward thesubject, N represents a normal vector from the subject, K represents adistance between the light source and the subject, S represents a vectorin the specular reflection direction, and V represents a directionvector from the subject toward the digital camera 100. In addition, β isa parameter that represents a shininess coefficient indicating thespread of the specular reflection light, and the larger β, the steeperthe specular reflection characteristics. As will be described later inrelation to parameter determination processing, the digital camera 100according to the present embodiment performs control such thatparameters of virtual light sources to be defined and variousreflectances of the subject under such light sources are dynamicallydetermined based on the analysis of the captured image.

Thus, the virtual light source addition processing unit 303 generatesoutput RGB signals reflecting a re-lighting result by adding the RGBsignals of the reflection components thus obtained and the RGB signalsderived from the de-gamma processing (R_(t), G_(t), B_(t)). That is,color components of the output RGB signals (R_(out), G_(out), B_(out))are obtained as follows: R_(out)=R_(t)+R_(a), G_(out)=G_(t)+G_(a), andB_(out)=B_(t)+B_(a).

A gamma processing unit 304 applies gamma correction processing to theoutput RGB signals generated by the virtual light source additionprocessing unit 303. After the application of the gamma correction, theoutput RGB signals are input to a luminance and chrominance signalconversion unit 305, converted into luminance and chrominance signals,and then output.

<<Parameter Determination Processing>>

Using a flowchart of FIG. 5, the following describes the specifics ofthe parameter determination processing that is executed by the digitalcamera 100 with the foregoing configuration according to the presentembodiment prior to the re-lighting processing. In the followingdescription, the parameter determination processing according to thepresent embodiment is executed prior to the re-lighting processing todetermine operational parameters for the blocks of the re-lightingprocessing unit 114 used in the execution of the re-lighting processing.However, the parameters are not limited to being set at this timing, andthe present parameter determination processing may be executed inparallel with the re-lighting processing. Processing corresponding tothe flowchart can be realized as the system control unit 50 reads out acorresponding processing program stored in, for example, the nonvolatilememory 121, extracts the processing program to the system memory 122,and executes the processing program. In the following description, thepresent parameter determination processing is started upon detection of,for example, issuance of a shooting instruction while a shooting modefor executing the re-lighting processing is set.

In step S501, the system control unit 50 determines whether a modecurrently set for re-lighting processing is a mode for manuallydetermining virtual light source parameters. The system control unit 50proceeds to step S504 if it determines that the current mode is the modefor manually determining the virtual light source parameters, and tostep S502 if it determines that the current mode is a different mode.

In step S502, the system control unit 50 obtains information ofevaluation values that have been generated by the evaluation valueobtainment unit 208 with respect to a face region in a captured image,and estimates the characteristics of an ambient light source.Specifically, based on the information of the evaluation values, thesystem control unit 50 estimates an illumination direction of theambient light source with respect to a subject and a diffusion degree oflight from the ambient light source. The estimation in this step isrealized by, for example, the following processing procedure.

First, the system control unit 50 determines whether a scene in whichthe captured image was obtained is a backlight scene. The system controlunit 50 determines that the scene is the backlight scene if thefollowing conditions are satisfied: the illuminance calculated based onexposure information at the time of image capture is higher than apredetermined threshold, and an upper region of the captured imageexhibits a distribution of high luminances, as in the case of FIG. 6Afor example. On the other hand, the system control unit 50 determinesthat the scene is not the backlight scene if the captured image does notsatisfy such conditions, as in the case of FIG. 6B for example. Itshould be easily understood that the determination of the backlightscene is not limited to being made using this method, and may be madeusing any other method.

Next, the system control unit 50 estimates a direction of illuminationby the ambient light source. This estimation may be made based on, forexample, a luminance distribution in a face region serving as a mainsubject. FIG. 6B exemplarily depicts a case in which a horizontalluminance distribution in a face region 601 (a horizontal change in anaverage luminance value of face region pixels on the same verticalcoordinate, or a horizontal luminance histogram) generally indicates alower average luminance value for a larger horizontal coordinate. Inthis case, the system control unit 50 estimates that the ambient lightsource is on the left as viewed from the digital camera 100 and emittinglight in a rightward direction in the captured image. Note that theresult of the estimation of the illumination direction may include adetermination result showing that the subject is illuminated from thefront based on a vertical luminance distribution, in addition to aleftward or rightward direction.

The system control unit 50 also estimates a degree of diffusion(diffusion degree) of light emitted by the ambient light source. Thesystem control unit 50 estimates the degree of diffusion because lightfrom the ambient light source may be incident on the subject not onlydirectly, but also through diffusion due to clouds, dust, and the likein and around a shooting environment. This estimation may be made basedon a contrast (a difference between the maximum and minimum luminancevalues) of the subject obtained from the luminance distribution in theface region, similarly to the case of the illumination direction. Inthis case, specifically, the system control unit 50 estimates thatintense light with a low diffusion degree is emitted if the contrast ishigher than a predetermined threshold, and that soft light with a highdiffusion degree is emitted if the contrast is lower than thepredetermined threshold.

In step S503, in accordance with the estimated characteristics of theambient light source, the system control unit 50 determines a positionat which a virtual light source is to be defined and variousreflectances of the subject illuminated by the virtual light source. Inthis step, the determination may be made in line with, for example,determination criteria of FIG. 7 that have been preset based on acombination of a diffusion degree and an illumination direction of theambient light.

FIG. 8A shows an exemplary case in which the estimated characteristicsof the ambient light source indicate that light with a low diffusiondegree is obliquely (in a leftward or rightward direction) illuminatinga subject (so-called oblique light). In this case, as intense(high-contrast) shadows are cast on the subject as shown in FIG. 9A, thesystem control unit 50 defines a virtual light source at a position 801that is horizontally symmetric with the ambient light source to mitigatethe shadows as shown in FIG. 8A. Furthermore, for the purpose ofreducing the shadows, the reflectances of the subject illuminated by thevirtual light source are determined such that only diffuse reflectionoccurs, specifically, a diffusion reflectance k_(d) of 1.0 and aspecular reflectance k_(s) of 0.0 are set. Therefore, by executing there-lighting processing using the virtual light source thus defined, theshadows cast on the subject are reduced in a resultant image as shown inFIG. 9B for example.

FIG. 8B shows an exemplary case in which the estimated characteristicsof the ambient light source indicate that light with a high diffusiondegree is obliquely illuminating a subject (oblique light). In thiscase, as high-contrast shadows are not cast on the subject due to thehigh diffusion degree as shown in FIG. 9C, the system control unit 50defines a virtual light source at a position 802 for obliqueillumination, similarly to the ambient light source shown in FIG. 8B, toexhibit a stereoscopic (dimensional) effect. In order to add luster andgloss to the low-contrast shadows for a natural stereoscopic effect, theposition 802 is determined such that the virtual light source and theambient light source illuminate the subject in the same direction.Furthermore, in order to represent luster and gloss, the reflectances ofthe subject illuminated by the virtual light source are determined suchthat only specular reflection occurs, specifically, a diffusionreflectance k_(d) of 0.0 and a specular reflectance k_(s) of 0.5 areset. In addition, a shininess coefficient β of 10.0 is set for thespecular reflection to represent relatively steep reflection. Therefore,by executing the re-lighting processing using the virtual light sourcethus defined, a supplementary representation of reflection is added tothe subject and the stereoscopic effect is enhanced as shown in FIG. 9Dfor example.

FIG. 8C shows an exemplary case in which the estimated characteristicsof the ambient light source indicate that a subject is illuminated bylight with a low diffusion degree from the front (so-called followlight). In this case, although a captured image gives an impression offlatness as shadows are not cast on the front of the subject, intenseshadows appear on some parts, such as the nose and chin, as shown inFIG. 9E. In view of this, the system control unit 50 defines a virtuallight source at a position 803 that is diagonal with respect to thesubject as shown in FIG. 8C to add a stereoscopic effect whilemitigating the intense shadows cast on some parts. Furthermore, for thepurpose of reducing the shadows and adding the stereoscopic effect, thereflectances of the subject illuminated by the virtual light source aredetermined such that two types of reflection characteristics areexhibited, specifically, a diffusion reflectance k_(d) of 0.5 and aspecular reflectance k_(s) of 0.8 are set. In addition, a shininesscoefficient β of 10.0 is set for specular reflection to representrelatively steep reflection. Therefore, by executing the re-lightingprocessing using the virtual light source thus defined, the shadows caston some parts of the subject are reduced and a representation ofreflection with a stereoscopic impression is added to the entire faceregion as shown in FIG. 9F for example.

FIG. 8D shows an exemplary case in which the ambient light source hasbeen estimated to emit backlight. In this case, as the ambient light isnot illuminating the front of a subject, a captured image gives animpression of flatness with the dark subject and a low contrast as shownin FIG. 9G. In view of this, the system control unit 50 defines avirtual light source for increasing the brightness of the subject at aposition 804, and defines a virtual light source for adding astereoscopic effect at a position 805. In order to prevent theappearance of an unnatural illumination effect in the backlight scene,the virtual light source for adding the stereoscopic effect is defined,for example, on the side of the subject or another position that can bedetermined to be equivalent in depth to the subject in a depthdirection. As shown in FIG. 8D, these two types of light sources withdifferent purposes are defined at either side of the horizontaldirection depending on the illumination direction of the ambient lightsource. More specifically, the system control unit 50 determines thehorizontal positions at which the virtual light sources are to bedefined such that the horizontal components of the estimatedillumination direction of the ambient light source have the samepositive or negative sign. Furthermore, as the virtual light sourceshave different purposes, values set as the reflectances of the subjectilluminated by one light source are different from values set as thereflectances of the subject illuminated by another light source. Thatis, the reflectances under illumination by the virtual light source forincreasing the brightness (at the position 804) are determined such thatonly diffusion reflection occurs, specifically, a diffusion reflectancek_(d) of 1.0 and a specular reflectance k_(s) of 0.0 are set. On theother hand, the reflectances under illumination by the virtual lightsource for adding the stereoscopic effect (at the position 805) aredetermined such that only specular reflection occurs, specifically, adiffusion reflectance k_(d) of 0.0 and a specular reflectance k_(s) of1.0 are set. Furthermore, a shininess coefficient β of 5.0 is set forthe specular reflection to represent reflection that adds a relativelymild highlight without giving an unnatural appearance to the backlightscene. Therefore, by executing the re-lighting processing using thevirtual light source thus defined, a supplementary representation ofreflection is added to the subject, the brightness is increased, and thestereoscopic effect is enhanced as shown in FIG. 9H for example.

If the system control unit 50 determines that the current mode is themode for manually determining the virtual light source parameters instep S501, it determines a position at which a virtual light source isto be defined and various reflectances of the subject illuminated by thevirtual light source based on a user's operational input in step S504.After making this determination, the system control unit 50 proceeds tostep S505.

In step S505, the system control unit 50 supplies information of variousvirtual light source parameters that have been determined to the virtuallight source reflection component calculation unit 311, configures thesettings to use the parameters in the re-lighting processing, andterminates the present parameter determination processing.

In the foregoing exemplary description of the present embodiment,diffuse reflection and specular reflection are used as models ofreflection on a subject under a virtual light source; however, models ofreflection are not limited to them, and any model of reflection may beused. For example, a model of reflection utilizing the bidirectionalreflectance distribution function (BRDF) and a model of reflectionutilizing a simulation of internal scattering inside a subject may beused, and parameters of the reflective characteristics of such modelsmay be controlled based on the way the ambient light reaches thesubject.

In the foregoing description of the present embodiment, an illuminationdirection and a diffusion degree of an ambient light source thatilluminates a subject are estimated as the characteristics of theambient light source; however, any information that identifies theilluminating condition of the subject, such as a type and position ofthe ambient light source, may be used as the characteristics of theambient light source.

In the foregoing description of the present embodiment, the reflectancesunder illumination by virtual light source are defined on a per-lightsource basis, and are constant regardless of a subject category;however, no limitation is intended in this regard. For example, for eachsubject category (e.g., a skin, hair, or clothes of a human), referencereflectances may be prestored, and values obtained by correcting thereference reflectances in accordance with the estimated characteristicsof an ambient light source may be set as the corresponding reflectances.This enables a reduction in the intensity of specular reflectioncomponents and other adjustments in, for example, portions of a subjectthat are not desired to exhibit specular reflection, thereby achieving amore favorable re-lighting result.

As described above, the image processing apparatus according to thepresent embodiment can generate an image to which a favorableillumination effect has been added by defining a virtual light sourcewith specific reflective characteristics during illumination based onthe estimated characteristics of an ambient light source in an imagecapture environment.

Second Embodiment

In the description of the foregoing embodiment, the characteristics ofan ambient light source estimated from a brightness distribution in acaptured image are used to determine a position of a virtual lightsource to be defined and the reflective characteristics of a subjectilluminated by the virtual light source. In the following description,the present embodiment adopts an approach to determining various virtuallight source parameters based on a subject type, brightness, a shootingscene, and the like to achieve a re-lighting result that is moredesirable for a photographer. It will be assumed that a functionalconfiguration of a digital camera 100 according to the presentembodiment is similar to a functional configuration of the digitalcamera 100 according to the first embodiment, and thus a descriptionthereof will be omitted.

<<Parameter Determination Processing>>

Using a flowchart of FIG. 10, the following describes the specifics ofparameter determination processing that is executed by the digitalcamera 100 according to the present embodiment prior to re-lightingprocessing. In the following description, the parameter determinationprocessing according to the present embodiment is similarly executedprior to the re-lighting processing to determine operational parametersfor the blocks of the re-lighting processing unit 114 used in theexecution of the re-lighting processing. However, the parameters are notlimited to being set at this timing, and the present parameterdetermination processing may be executed in parallel with there-lighting processing. Processing corresponding to the presentflowchart can be realized as the system control unit 50 reads out acorresponding processing program stored in, for example, the nonvolatilememory 121, extracts the processing program to the system memory 122,and executes the processing program. In the following description, thepresent parameter determination processing is started upon detection of,for example, issuance of a shooting instruction while a shooting modefor executing the re-lighting processing is set.

In order to identify a subject type and a shooting scene, the parameterdetermination processing according to the present embodiment usesinformation of a shooting mode that was set when a captured image wasobtained. In the following description of the processing, a portraitmode, a food (shooting) mode, and a flower/plant mode with distinctsubject types are used as examples of modes with easy-to-identifysubject types as shown in FIG. 11; however, this should not restrict theuse of other shooting modes. In other shooting modes, the processing maybe executed such that the characteristics of a subject are taken intoconsideration, or preferences of an image editor are reflected, indetermining various virtual light source parameters. For example, in abacklight portrait mode, an evening scene portrait mode, and the like,an illumination direction of an ambient light source and a main subjecttype can be identified, and thus can be taken into consideration indetermining various virtual light source parameters in the followingprocessing.

In step S1001, the system control unit 50 obtains information of asubject generated by the evaluation value obtainment unit 208 inrelation to a captured image, and determines a main subject region basedon the information. The main subject region may be determined using anymethod, for example, a method in which a human region detected by theface detection unit 113 is used as the main subject region, or a methodin which a region having the same tints as a subject focused in thevicinity of the center of the captured image is used as the main subjectregion. The system control unit 50 also calculates information of theaverage brightness of the determined main subject region.

In step S1002, the system control unit 50 obtains information indicatinga shooting mode that was set when the captured image was obtained, anddetermines various virtual light source parameters to be used in there-lighting processing based on the information indicating the shootingmode and the brightness of the main subject region. That is, similarlyto the first embodiment, the system control unit 50 determines aposition of a virtual light source to be defined and the reflectances ofthe subject illuminated by the light source. In this step, thedetermination may be made in line with, for example, determinationcriteria of FIG. 11 that have been preset based on a combination of ashooting mode that was set and the brightness of the main subjectregion.

FIG. 12A shows an exemplary case in which the shooting mode is theportrait mode and the average brightness of a human region serving as amain subject region has been determined to be lower than a predeterminedthreshold, that is, the human region has been determined to be dark. Inthis case, a virtual light source 1201 for brightening the main subjectregion is defined at a position that is diagonally in front of a mainsubject (smaller in depth than the main subject in a depth direction) soas to obliquely illuminate the main subject. Furthermore, thereflectances of the subject illuminated by the virtual light source 1201are set such that light is reflected only by diffuse reflection,specifically, a diffusion reflectance k_(d) of 1.0 and a specularreflectance k_(s) of 0.0 are set. In addition, as the light reflected bythe diffuse reflection alone exhibits a poor stereoscopic effect, avirtual light source 1202 for adding a stereoscopic effect is defined toilluminate the main subject from the side. The reflectances of thesubject illuminated by the virtual light source 1202 are set such thatlight is reflected only by specular reflection, specifically, adiffusion reflectance k_(d) of 0.0 and a specular reflectance k_(s) of0.5 are set. Furthermore, a shininess coefficient β of 5.0 is set forthe specular reflection to exhibit the stereoscopic effect with a softhighlight. In this way, parameters that yield the re-lighting resultshown in FIG. 9H can be determined without determining a backlightscene.

In a case in which the shooting mode is the portrait mode and theaverage brightness of the human region serving as the main subjectregion has been determined to be higher than the predeterminedthreshold, that is, the human region has been determined to be bright,only the virtual light source for adding the stereoscopic effect isdefined. For example, it is sufficient to define only the virtual lightsource 1202 for the specular reflection shown in FIG. 12A.

FIG. 12B shows an exemplary case in which the shooting mode is the foodmode and the average brightness of an image region of food serving as amain subject has been determined to be lower than a predeterminedthreshold, that is, the image region has been determined to be dark. Inthis case, a virtual light source 1203 for brightening a main subjectregion is defined at a position above the main subject so as toilluminate the main subject from above. Furthermore, the reflectances ofthe subject illuminated by the virtual light source 1203 are set suchthat light is reflected only by diffuse reflection, specifically, adiffusion reflectance k_(d) of 1.0 and a specular reflectance k_(s) of0.0 are set. In addition, a virtual light source 1204 for adding lusterand a stereoscopic effect to the food is defined at a position that isdiagonally behind the main subject (larger in depth than the mainsubject in a depth direction) so as to facilitate the appearance of theeffect of specular reflection. The reflectances of the subjectilluminated by the virtual light source 1204 are set such that light isreflected primarily by the specular reflection, specifically, adiffusion reflectance k_(d) of 0.2 and a specular reflectance k_(s) of1.0 are set. A shininess coefficient β of 20.0 is set for the specularreflection to exhibit the stereoscopic effect with a relatively steephighlight.

In a case in which the shooting mode is the food mode and the averagebrightness of the image region of the food serving as the main subjecthas been determined to be higher than the predetermined threshold, thatis, the image region has been determined to be bright, only the virtuallight source for adding luster and the stereoscopic effect is defined.For example, it is sufficient to define only the virtual light source1204 for the same purpose shown in FIG. 12B.

FIG. 12C shows an exemplary case in which the shooting mode is theflower/plant mode and the average brightness of an image region offlowers serving as a main subject has been determined to be lower than apredetermined threshold, that is, the image region has been determinedto be dark. In this case, a virtual light source 1205 for brightening amain subject region is defined at a position that is diagonally in frontof the main subject so as to obliquely illuminate the main subject. Thereflectances of the subject illuminated by the virtual light source 1205are set such that light is reflected primarily by diffuse reflection,specifically, a diffusion reflectance k_(d) of 1.0 and a specularreflectance k_(s) of 0.2 are set. In the case of the flower/plant mode,a shininess coefficient β of 3.0 is set for specular reflection toproduce a gradual highlight, because defining the virtual light sourceto emphasize luster can yield an image with an unnatural impressionthrough re-lighting. For a similar reason, a virtual light source formainly producing specular reflection components is not defined in theflower/plant mode.

In a case in which the shooting mode is the flower/plant mode and theaverage brightness of an image region of plants serving as a mainsubject has been determined to be higher than the predeterminedthreshold, that is, the image region has been determined to be bright,no virtual light source is defined because it is unnecessary to executethe re-lighting processing using a virtual light source. That is,control is performed such that the re-lighting processing or processingof the virtual light source reflection component calculation unit 311 isnot executed.

In step S1003, the system control unit 50 supplies information ofvarious virtual light source parameters that have been determined to thevirtual light source reflection component calculation unit 311,configures the settings to use the parameters in the re-lightingprocessing, and terminates the present parameter determinationprocessing.

In the foregoing description of the present embodiment, various virtuallight source parameters are determined based on a shooting mode and theaverage brightness of a main subject region; however, an item that isused in combination with a shooting mode to make this determination isnot limited to the average brightness. For example, various virtuallight source parameters may be determined by combining information of anillumination direction of an ambient light source used in theabove-described first embodiment, or various virtual light sourceparameters may be determined based solely on a shooting mode.

Third Embodiment

As briefly mentioned in the first embodiment, when a subject is a human,for example, actual reflective characteristics of the subject vary witheach subject category (e.g., a skin, hair, or clothes), and it is thuspreferable that reflectances of a subject illuminated by a definedvirtual light source vary with each subject category. On the other hand,for example, an illumination direction and other conditions for addingfavorable effects vary with each part of a human face as follows:so-called catch light that causes the pupils to reflect light iscompatible with the eyes, light that adds luster to the lips iscompatible with the mouth, and light that enhances a stereoscopic effecton the nasal bridge is compatible with the nose. In the followingdescription, the present embodiment adopts an approach to achieving are-lighting result by defining a virtual light source such thatillumination conditions and reflectances of a subject vary with eachsubject category specified. It will be assumed that a functionalconfiguration of a digital camera 100 according to the presentembodiment is similar to a functional configuration of the digitalcamera 100 according to the first embodiment, and thus a descriptionthereof will be omitted.

<<Parameter Determination Processing>>

Using a flowchart of FIG. 13, the following describes the specifics ofparameter determination processing that is executed by the digitalcamera 100 according to the present embodiment prior to re-lightingprocessing. In the following description, the parameter determinationprocessing according to the present embodiment is similarly executedprior to the re-lighting processing to determine operational parametersfor the blocks of the re-lighting processing unit 114 used in theexecution of the re-lighting processing. However, the parameters are notlimited to being set at this timing, and the present parameterdetermination processing may be executed in parallel with there-lighting processing. Processing corresponding to the presentflowchart can be realized as the system control unit 50 reads out acorresponding processing program stored in, for example, the nonvolatilememory 121, extracts the processing program to the system memory 122,and executes the processing program. In the following description, thepresent parameter determination processing is started upon detection of,for example, issuance of a shooting instruction while a shooting modefor executing the re-lighting processing is set.

In step S1301, the system control unit 50 obtains information of theresult of detection by the face detection unit 113 with respect to aface region in a captured image, and identifies regions of predeterminedsubject categories in the face region. Specifically, based on the resultof detection, the system control unit 50 identifies the position of theface region and the positions of regions corresponding to the eyes,mouth, and nose in the face region.

In step S1302, for each region that has been positionally identified instep S1301, the system control unit 50 determines various virtual lightsource parameters corresponding to a subject category of the region.That is, for each part (the eyes, mouth, nose, or another region) of amain subject region including the face region, the system control unit50 determines various virtual light source parameters that add anillumination effect in the re-lighting processing.

For example, with respect to a nose region, the system control unit 50defines a virtual light source 1401 that illuminates the nose regionfrom a position that is slightly displaced sideways from the front ofthe face (oblique illumination) to enhance the stereoscopic effect onthe nasal bridge as shown in FIG. 14A. Furthermore, to enhance thestereoscopic effect on the nasal bridge, the reflectances of the noseregion illuminated by the virtual light source 1401 are set such thatlight is reflected only by specular reflection, specifically, adiffusion reflectance k_(d) of 0.0 and a specular reflectance k_(s) of0.8 are set. In addition, a shininess coefficient β of 10.0 is set forthe specular reflection to exhibit the stereoscopic effect with arelatively steep highlight.

For example, with respect to a mouth region, the system control unit 50defines a virtual light source 1402 that illuminates the mouth regionusing oblique light from the vicinity of the side of the mouth toenhance luster of the lips as shown in FIG. 14A. Furthermore, to enhanceluster of the lips, the reflectances of the mouth region illuminated bythe virtual light source 1402 are set such that light is reflected onlyby specular reflection, specifically, a diffusion reflectance k_(d) of0.0 and a specular reflectance k_(s) of 1.0 are set. In addition, ashininess coefficient β of 30.0 is set for the specular reflection toproduce a steep highlight and exhibit luster of the lips.

For example, with respect to an eye region, the system control unit 50defines a virtual light source 1403 that illuminates the eye region froma position that is displaced downward from the front of the face topresent shininess in the eyes by causing the pupils to reflect light asshown in FIG. 14A. Furthermore, to exhibit shininess in the eyes, thereflectances of the eye region illuminated by the virtual light source1503 are set such that light is reflected only by specular reflection,specifically, a diffusion reflectance k_(d) of 0.0 and a specularreflectance k_(s) of 1.0 are set. In addition, a shininess coefficient βof 50.0 is set for the specular reflection to produce a steep highlightand exhibit a catch light effect.

For example, with respect to other regions, the system control unit 50defines a virtual light source 1404 that obliquely illuminates the mainsubject region as shown in FIG. 14A. Furthermore, the reflectances ofother regions illuminated by the virtual light source 1404 are set suchthat light is reflected primarily by specular reflection, specifically,a diffusion reflectance k_(d) of 0.2 and a specular reflectance k_(s) of0.8 are set. In addition, a shininess coefficient β of 5.0 is set forthe specular reflection to produce a gradual highlight and exhibit anoverall stereoscopic effect.

In step S1303, the system control unit 50 supplies information ofvarious virtual light source parameters that have been determined to thevirtual light source reflection component calculation unit 311,configures the settings to use the parameters in the re-lightingprocessing, and terminates the present parameter determinationprocessing. At this time, in order to prevent the virtual light sourcesdefined for specific regions from adding an illumination effect to otherregions, the virtual light source reflection component calculation unit311 performs control such that the two types of reflectances are bothset to 0.0 for non-target regions. In this way, favorable effects areadded to parts of the subject on a part-by-part basis as a result ofre-lighting as shown in FIG. 14B.

In the foregoing description of the present embodiment, various virtuallight source parameters determined for the eye, nose, and mouth regionsin a face region of a human differ from those for other regions in theface region; however, the present invention is not limited to beingembodied in this way. For example, a virtual light source used to add anillumination effect may vary with each specific subject category (e.g.,an object, an animal, or food) rather than each specific human part.Therefore, provided that the re-lighting processing is applied to a mainsubject region, the invention according to the present embodiment can beapplied as long as virtual light sources with different types ofparameters are defined for different categories of subjects included inthe main subject region.

In the foregoing description of the second and third embodiment, thereflective characteristics of a subject illuminated by a virtual lightsource are determined fundamentally based on the illuminating conditionby an ambient light source, and further in consideration of a shootingmode used in shooting and a subject category. However, in embodying thepresent invention, the illuminating condition by an ambient light sourceneed not necessarily be taken into consideration to determine thereflective characteristics, and the reflective characteristics may bedetermined based on a shooting mode used in shooting or a subjectcategory.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

1. (canceled)
 2. An image processing apparatus, comprising: a processor;and a memory including instructions that, when executed by theprocessor, cause the processor to function as: an obtaining unitconfigured to obtain an image derived from image capture; and an imageprocessing unit configured to add an effect of a virtual light on asubject included in the image obtained by the obtaining unit, whereinthe image processing unit determines the effect of the virtual lightbased on an illuminating condition of the subject by an ambient light,and the effect of the virtual light includes effects of virtual lightsilluminated by a plurality of light sources including a first lightsource and a second light source.
 3. The image processing apparatusaccording to claim 2, wherein the first light source is a virtual lightsource for increasing a brightness of the subject, and the second lightsource is a virtual light source for adding a stereoscopic effect to thesubject.
 4. The image processing apparatus according to claim 3, whereinthe virtual light source for increasing the brightness of the subjectadds an effect caused by a diffusion reflection to the subject.
 5. Theimage processing apparatus according to claim 3, wherein the virtuallight source for adding a stereoscopic effect to the subject adds aneffect caused by a specular reflection to the subject.
 6. The imageprocessing apparatus according to claim 2, wherein illuminationdirections of the first light source and the second light source aredifferent.
 7. The image processing apparatus according to claim 2,wherein the illuminating condition of the subject by the ambient lightis based on a diffusion degree of an ambient light source.
 8. The imageprocessing apparatus according to claim 2, wherein the illuminatingcondition of the subject by the ambient light is based on a contrast ofa subject region in the obtained image.
 9. The image processingapparatus according to claim 2, wherein the illuminating condition ofthe subject by the ambient light is based on a luminance distribution ina subject region of the obtained image.
 10. The image processingapparatus according to claim 2, wherein the illuminating condition ofthe subject by the ambient light is based on an illumination directionof the ambient light.
 11. The image processing apparatus according toclaim 2, wherein in a case where the illuminating condition of thesubject by the ambient light indicates a backlight, the image processingunit adds the effects of the virtual lights illuminated by the pluralityof light sources including the light source and the second light source.12. The image processing apparatus according to claim 2, wherein thevirtual lights illuminated by the first light source and the secondlight source have parameters for adding effects caused by a diffusionreflection and a specular reflection, and the parameters arerespectively defined.
 13. A control method for an image processingapparatus, the control method comprising: obtaining an image derivedfrom image capture; and adding an effect of a virtual light on a subjectincluded in the obtained image, wherein the effect of the virtual lightis determined based on illuminating condition of the subject by anambient light, and the effect of the virtual light includes effects ofvirtual lights illuminated by a plurality of light sources including afirst light source and a second light source.
 14. A computer readablestorage medium storing a program to be used by an image processingapparatus, the program being configured to cause a computer of the imageprocessing apparatus to execute: obtaining an image derived from imagecapture; and adding an effect of a virtual light on a subject includedin the obtained image, wherein the effect of the virtual light isdetermined based on illuminating condition of the subject by an ambientlight, and the effect of the virtual light includes effects of virtuallights illuminated by a plurality of light sources including a firstlight source and a second light source.